Virtual Machine Introspection Based Malware Behavior Profiling and Family Grouping

The proliferation of malwares have been attributed to the alternations of the original malware source codes. The malwares alternated from the same origin share some intrinsic behaviors and form a malware family. Expediently, identifying its malware family when a malware is first seen can provide useful clues to mitigating the threat. In this paper, a malware profiler (VMP) is proposed to profile the execution behaviors of a malware at the runtime by leveraging the virtual machine introspection (VMI) technique. The VMP inserts a plug-in inside the virtual machine monitor (VMM) to record the invoked Windows API calls with the parameters and return values as the profile of a malware. Based on the profiles, we then adopt a distance measurement and a phylogenetic tree construction method to discover the malware behavior groups. As expected, our study shows the malwares from a malware family are similar to each other and distinct from other malware families as well as the benign software. We then examines the goodness of the family grouping method of the VMP against existing anti-malware detection engines and some well-known grouping methods. We propose a novel peer voting method for evaluating the result of family grouping and the evaluation shows VMP is better than almost all of the compared anti-malware engines. At last, we establish a malware profiling website based on the proposed VMP for the public use.